Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation

ABSTRACT

A result higher resolution (HR) image of a scene given multiple, observed lower resolution (LR) images of the scene is computed using a Bayesian estimation image reconstruction methodology. The methodology yields the result HR image based on a Likelihood probability function that implements a model for the formation of LR images in the presence of noise. This noise is modeled by a probabilistic, non-Gaussian, robust function. Other embodiments are also described and claimed.

BACKGROUND

An embodiment of the invention is directed to signal processingtechniques to obtain a higher resolution, HR, image (or sequence ofimages) from multiple observed lower resolution images. Otherembodiments are also described.

In most electronic imaging applications, images with higher resolutionare generally more desirable. These are images that have greater pixeldensity and hence show greater detail than lower resolution images ofthe same scene. HR images have many applications, including medicalimaging, satellite imaging, and computer vision.

An HR image may be obtained by simply increasing the number and/ordensity of pixel sensor elements in the electronic image sensor chipthat is used to capture the image. This, however, may increase the sizeof the chip so much that capacitance effects will hamper the rapidtransfer of pixel signal values, thereby causing difficulty forobtaining high-speed captures and video. Another possibility is toreduce the physical size of each pixel sensor element; however, doing somay increase the noise level in the resulting pixel signal value.Additionally, increasing the number of pixel sensor elements increasesthe cost of the device, which in many situations is undesirable (e.g.,cameras mounted on mobile devices whose primary function is not imageacquisition, like personal digital assistants (PDA) and cellularphones), and in others is prohibitive (e.g., infrared sensors).Therefore, another approach to obtaining HR images (that need not modifythe lower resolution sensor) is to perform digital signal processingupon multiple lower resolution (LR) images captured by the sensor, toenhance resolution (also referred to as super resolution, SR, imagereconstruction).

With SR image reconstruction, multiple observed LR images or frames of ascene have been obtained that in effect are different “looks” of thesame scene. These may be obtained using the same camera, for example,while introducing small, so-called sub-pixel shifts in the cameralocation from frame to frame, or capturing a small amount of motion inthe scene. Alternatively, the LR images may be captured using differentcameras aimed at the same scene. A “result” HR image is thenreconstructed by aligning and combining properly the LR images, so thatadditional information, e.g. an increase in resolution or de-aliasing,is obtained for the result HR image. The process may also include imagerestoration, where de-blurring and de-noising operations are performedas well, to yield an even higher quality result HR image.

The reconstruction of the result HR image, however, is a difficultproblem because it belongs to the class of inverse, ill-posedmathematical problems. The needed signal processing may be interpretedas being the reverse of a so-called observation model, which is amathematically deterministic way to describe the formation of LR imagesof a scene (based upon known camera parameters). Since the scene isapproximated by an acceptable quality HR image of it, the observationmodel is usually defined as relating an HR discrete image of the scene(with a given resolution and pixel grid) to its corresponding LR images.This relationship (which may apply to the formation of both still imagesand video) may be given as the concatenation of a geometric transform, ablur operator, and a down-sampling operator, plus an additive noiseterm. Examples of the geometric transform include, global or localtranslation and rotation, while the blur operator attempts to duplicatecamera non-idealities, such as out of focus, diffraction limits,aberration, slow motion blur, and image sensor integration on a spatialregion (sometimes combined all together in a point spread function). Thedown-sampling operator down samples the HR image into aliased, lowerresolution images. This observation model may be expressed by themathematical relationshipY═W*f+n,  (1)where Y is the set of observed LR images and W represents the lineartransformation of HR pixels in an HR image f to the LR pixels in Y(including the effect of down-sampling, geometric transform and blur).The n represents additive noise having random characteristics, which mayrepresent, for example, the variation (or error) between LR images thathave been captured by the same camera without any changes in the sceneand without any changes to camera or lighting settings. Based on theobservation model in Equation (1), SR image reconstruction estimates theHR image f that corresponds to a given set of LR images Y.

A Bayesian estimation process (also referred to as stochastic orprobabilistic SR image reconstruction) may be used to estimate f, to getthe “result” HR image mentioned above. In that case, an “a posteriori”probability function (typically, a probability density function) ismathematically defined as p(f|Y), which is the probability of aparticular HR image f given the set of observed LR images Y. Applying amathematical manipulation, known as Bayes Law, the optimization problem,which is finding a suitable HR image f, e.g. one that has the highestprobability given a set of LR images or that maximizes p(f|Y), may bere-written asP(f|Y)=p(Y|f)*p(f),  (2)where p(f) is called the “Prior” probability density function that givesthe probabilities of a particular HR image prior to any observation. ThePrior indicates what HR images are more probable to occur based on, forexample, a statistical characterization of an ensemble of different HRimages. The Prior probability may be a joint probability, defined overall of the pixels in an HR image, and should be based on statisticaldata from a large number of images. However, estimating and describingthe Prior probability as a joint distribution over all pixels may not becomputationally feasible. Accordingly existing methods use approximatemodels, based on the fact that in many types of images, correlationsamong pixels decay relatively quickly with pixel distance. For example,the Prior may be based on a probabilistic construct called Markov RandomFields (MRFs). Rather than take the position that all HR images areequally likely, the MRF is tailored to indicate for example that certainpixel patterns (e.g., piece-wise continuous; text images) are morelikely than others. An image may be assumed to be globally smooth in amathematical sense, so the MRF typically used to define the Prior has anormal (Gaussian) probability distribution.

As to p(Y|f), that is called the “Likelihood” function; it is aprobability density function that defines the probabilities of observingLR images that would correspond to a particular HR image. The Likelihoodmay be determined based on the observation model described above by themathematical relationship in Equation (1), where the noise term istypically assumed to have a Gaussian probability distribution. Theestimation process becomes one of iteratively determining trial HRimages and stopping when there is convergence, which may signify that amaximum of the a posteriori probability function has been reached.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example andnot by way of limitation in the figures of the accompanying drawings inwhich like references indicate similar elements. It should be noted thatreferences to “an” embodiment of the invention in this disclosure arenot necessarily to the same embodiment, and they mean at least one.

FIG. 1 is a graph of robust and normal probability densities.

FIG. 2 is a graph of Likelihood and Prior probability functions for atrial HR image.

FIG. 3 is a flow diagram of some of the operations in a super resolutionimage reconstruction process.

FIG. 4 is a flow diagram of some of the operations in a super resolutionimage reconstruction method operating on color images.

FIGS. 5 and 6 shows two images that illustrate the results of applyingthe super resolution method to webcam images.

FIGS. 7-11 shows images that illustrate the results of applying thesuper resolution method to images from a scanning beam nano-imagingdevice.

DETAILED DESCRIPTION

An embodiment of the invention is a method for image processing in whicha Bayesian estimation image reconstruction methodology computes a resultHR image of a scene given multiple observed LR images. The result HRimage is based on a Likelihood probability function that implements anobservation model for the formation of LR images in the presence ofnoise. The methodology models the noise by a probabilistic,non-Gaussian, robust function. Such robust functions are defined in thestatistical estimation literature and are characterized by long tails inthe probability density function, as shown in FIG. 1. In contrast to thenormal or Gaussian distribution, the robust distribution acknowledgesthe occurrence of a few points that are affected by an unusually highamount of noise, also referred to as outliers (which are at the tailends of the density graphs shown in FIG. 1). This change to the modelingof noise better models the formation of LR images from the HR image, sothat the method produces a more accurate solution. Thus, althoughimplementing the SR process is made easier when the noise is modeled bya Gaussian probability function, such an assumption does not adequatelyhandle images that contain different levels of outliers, which arecommon in SR reconstruction, due especially to inaccuracies in the imagealignment.

Referring now to FIG. 2, a graph of probability density for a trial HRimage is shown in which the example Likelihood and Prior function havebeen drawn. The maximum a posteriori (MAP) is proportional to the Priorand the Likelihood as given by Equation (2) above. In this caseLikelihoods for two different assumed noise distributions (R) and (G)are shown, corresponding respectively to a robust probability functionto model the noise (R), and another using a normal or Gaussian (G). Thegraph illustrates the effect of an outlier in a given LR image (notshown) that translates into a dip in the Likelihood (G) for certainareas of a trial HR image. This strong dip in the Likelihood (G) is dueto the outlier dominating the Likelihood function, indicating arelatively low probability for the set of observed LR images, given thisparticular trial HR image. However, in actuality, it may be that thetrial HR image is a good one, and that the only reason why theLikelihood value is low is due to the outlier (in one or more of theobserved LR images). This domination of the Likelihood function by anoutlier is negated by the use of a robust function which downplays therole of outlier pixels in observed LR images. Accordingly, the computedrobust Likelihood (R) for this particular set of observed LR images(given that trial HR image) is higher than if the noise was modeled by aGaussian function.

The various embodiments of the invention described here may prove therobustness of the SR process such that it can be used in different typesof real world applications to be described below. FIG. 3 illustrates aflow diagram of some of the operations in a SR method. The methodcontains a main loop that is repeatedly performed as part of aniterative process to determine the result (or final) HR image 104. Thisprocess may attempt to find an optimum value, here a minimum, for anerror function E. More specifically, this error function may be definedas the negative logarithm of the posterior probability in Equation (2).This error function may be minimized using any standard minimizationtechniques. For example, FIG. 3 shows the use of the conjugate gradientmethod which is an iterative method that provides an acceptable balancebetween complexity and speed of convergence. The criteria forconvergence is ΔE<T, which tests whether the error or difference in theposterior probability of Equation (2), between two successive trial HRimages, is less than a predefined threshold, T (block 106). Analternative test is to define ΔE as a difference between consecutivetrial HR images.

The conjugate gradient method computes the gradient of the errorfunction which has two terms in this embodiment, one corresponding tothe Likelihood and the other to the Prior. The computation of theLikelihood gradient (block 108) involves the application of standardimage processing operations including geometric warping, linearfiltering, and subsampling/upsampling, for example, that model both theforward and the reverse of the LR image formation process. To computethe Likelihood gradient, an initial, trial HR image is needed. This maybe, for example, a combination of one or more of an input (observed) LRimage sequence (block 110) that have been aligned (block 114) to yieldan HR image with an initial alignment (block 116). The results of thisinitial alignment are then used to compute the Likelihood gradient(block 108). Recall once again that the SR method assumes that the inputLR images are the result of resampling an HR image, and the goal is tofind the HR image which, when resampled in the grid of the input LRimages according to the imaging observation model, predicts well theinput (observed) LR images.

The other half of the main computation loop in FIG. 3 is concerned withthe Prior gradient (block 120). Different types of probability functionsmay be used for the Prior, but in the case of a robust MRF, the Priorgradient is equivalent to one update of a corresponding robustanisotropic diffusion filter, as described in Michael J. Black, et al.,“Robust Anisotropic Diffusion”, Institute of Electrical and ElectronicsEngineers, IEEE Transactions on Image Processing, Vol. 7, No. 3, March1998. Other implementations of the Prior function and its correspondinggradient may alternatively be used.

The gradients computed in blocks 108 and 120 indicate to the iterativeprocess the direction in which to move so as to come closer to a peak ortrough in the combination of the Likelihood and Prior functions (seeFIG. 2). This movement along the plots of the Likelihood and Priorfunctions results in a change or update (block 124) to the next HRimage, which generates the current, trial HR image 126. This currenttrial HR image 126 is then inserted into Equation (2) and a ΔE, which isthe difference between the current value of Equation (2) and a previousvalue of Equation (2) is compared to a threshold T (block 106). If theΔE is still too high, then the gradient computation loop is repeated. Anadditional decision may be made as to whether or not a refinement of theLR image initial alignment (block 116) is needed, in block 128. Thisalignment may be evaluated using any one of conventional techniques.Operation may then proceed with an alignment of the LR images to a newHR image (block 130) resulting in a refined alignment (block 134). Thenext gradient computation for the Likelihood may use an HR image thathas this refined alignment 134.

Note that if a normal or Gaussian function is assigned to model theadditive noise for computing the Likelihood (and its gradient), then theHR image update (block 124) may cause the next trial HR image 126 to bechanged too much, due to an outlier in the input LR image sequence 110,thereby causing the methodology to select a less optimal final HR image104.

A methodology for using the robust functions to model the noise in theobservation model, which functions are able to “down weight” or in somecases essentially ignore outliers in the SR process, may be as follows.Ideally, the probability distribution of the noise should be learnedgiven a set of training examples consisting of HR images and theircorresponding LR images. This set can be difficult to obtain, and evenif it is available, it might not contain the noise attributed to errorsin the alignment. For this reason, in most cases it may be better to usea generic robust function from the statistics literature. The choice ofthe robust function to use might depend on the knowledge available aboutthe current images. For example, the process may use one of twodifferent robust functions depending on the available knowledge aboutthe presence of outliers. If it is expected that the observed LR imageswill have relatively few outliers, then the robust function used tomodel the additive noise may be the well known Huber function. Note thatsuch outliers may be caused by alignment errors, inaccurate modeling ofblur, random noise, moving objects, motion blur, as well as othersources. Thus, if a process is expected to have, for example, relativelyaccurate image alignment, the Huber function may be used to model theadditive noise. The Huber function, although not being extremely robust,has the advantage of being convex, thus essentially guaranteeing aunique optimum (maximum or minimum) in the Likelihood function.

On the other hand, if it is expected that the observed LR images willhave relatively many outliers (e.g., salt and pepper noise, and/orregions in the aligned image that have inaccurate alignment), the robustfunction may be set to a Tukey function which is considered very robust,thereby essentially eliminating any effect of the outliers in thesolution.

In addition to the option of setting the robust function to be adifferent one depending on whether relatively few or many outliers areexpected, a shape of the robust function may be estimated and alteredaccording to the availability of training data. For example, the shapeof the robust function may be adjusted by a scale factor, where if thereis sufficient training data in the form of one or more ground truth HRimages and their corresponding LR images, the scale factor is estimatedfrom samples obtained in computing an error between the observed LRimages of the scene and their projections from the ground truth HRimages.

On the other hand, if there is no such training data, the scale factormay be estimated by taking a current, trial HR image 126 (FIG. 3) as aground truth HR image, and applying a robust estimator as the scalefactor. This robust estimator may be, for example, the median ofresiduals with respect to the median value. Other types of robustestimators may alternatively be used here.

According to another embodiment of the invention, the Prior function maybe as follows. If there is specific or statistical informationconcerning the expected HR images, such as computer aided design (CAD)models for structures captured in the observed LR images, thenprocedures similar to those described in U.S. patent application Ser.No. 10/685,867 entitled “Model Based De-Noising of Images and ImageSequences”, assigned to the same Assignee as that of this patentapplication, may be used. Those procedures may be particularlybeneficial in applications such as microscopic imaging of siliconstructures using scanning methods (e.g., focused ion beam; scanningelectron microscope). That is because the structures being imaged inthat case have corresponding, underlying CAD models.

On the other hand, if no such model-based knowledge of the expected HRimages exists, then a generic Prior function in the form of, forexample, a robust MRF may be used. The portion of the gradient thatcorresponds to such a Prior is equivalent to one update of ananisotropic diffusion methodology. For this reason, any one of severaldifferent anisotropic diffusion methods that best adapts to the type ofimages that are to be expected may be used. For generic images, however,a good option for preserving edges in detail in the image is the Tukeyfunction on a 4-neighbor MRF, as described by Black, et al., the articleidentified above. Other options include neighbor schemes (e.g.,8-neighbor) with cost functions that are adapted to the type of filterbeing used, that can be generic or learned from a training set ofimages. See also H. Scharr, et al. “Image Statistics and AnisotropicDiffusion”, IEEE Conference on Computer Vision and Pattern Recognition,Pages 840-847, Oct. 13-16, 2003. Use of either of the above options inthe SR methods described here is expected to provide improvedperformance relative to the use of a Gaussian MRF as the generic Prior.

Image Alignment

In the previous discussion, it may be assumed that the geometrictransformations that align the sampling grids of the observed or inputLR image sequence 110 with the sampling grid of the HR image 126 wereknown. However, in most cases, this information is not known a priori,unless the LR image sequence has been obtained under explicit controlledmotion of the image acquisition device relative to the objects in thescene. Therefore, an estimate of these geometrical transforms are oftenneeded. According to another embodiment of the invention, thesegeometrical transforms may be estimated as follows.

First, an initial estimate of the geometric transforms between theobserved or input LR images is obtained. Different options may be usedhere, depending on the characteristics of the motion of the imageacquisition device relative to the scene being imaged. For genericsequences, with small changes in perspective, a global affinetransformation model is used. For images with large changes inperspective, the affine model may be no longer appropriate so thathigher order models (e.g., projective) should be used. Finally, if thereis relative motion between the objects in the scene or perspectivechanges together with discontinuities in depth, global models maygenerally not be appropriate, such that either a dense local motionmodel (optical flow) or a layered model should be used.

Once a reasonable estimate of the HR image has been obtained (forexample after 4-6 iterations), the initial alignment 116 (FIG. 3) may berefined (block 134) using the current version of the trial HR image 126.The latter is expected to provide more accurate results than the LR toLR image alignment 114, because the LR images are affected by aliasing.This technique may be compared to a combined Bayesian estimation forboth the HR image and the geometrical transform.

Regardless of the motion model used for the alignment, as well as thetype of alignment (that is LR to LR, or HR to HR), state of the artgradient based, multi-resolution, robust image motion estimation methodsshould be used to determine the alignment that will be input into theLikelihood gradient computation block 108 (FIG. 3).

Color Images

The embodiments of the invention described above may be assumed tooperate with gray-level images. These SR methods, however, may also beapplied to color images, which are usually presented as three componentsfor each pixel, corresponding to Red (R), Green (G) and Blue (B) colorsbands. The method can be applied to each color band independently toobtain a final HR image in RGB. However, applying the method to thethree RGB bands is very computationally demanding. For this reason analternative method is described in the flow diagram shown in FIG. 4,which is less computationally intensive, and produces results that areperceptually equivalent to applying the method to all three color bands.In this embodiment, operation begins with converting the input LR colorimage sequence 404 from the RGB color space into a color space that isconsistent with the human perception of color, in this case CIELab(Commite Internationale de l'Eclairage) (block 408). In the CIELab colorspace, the three components are luminance (L) and two opponent colorcomponents (a, b). The SR methodology described above is applied only tothe L component sequence 412, rather than the a, b components 416,because the human visual system detects high spatial frequencies mostlyon luminance, and not in the opponent color components. Therefore, forthe a, b opponent color components 416, the reconstruction to obtain HRa, b images 422 may be simply taking the average of aligned LR images(block 417), where this operation helps reduce noise in the componentimages, and then interpolating to match the needed HR image resolutionusing standard interpolation methods, such as bilinear interpolation(block 418). This methodology is much faster than applying the SR method414 to all three color channels, and it is expected to be perceptuallythe same, in most cases. A conversion back to RGB color components(block 430) is performed to obtain the result HR color image 432 in theconventional RGB space.

The methodology of FIG. 4 has been implemented and applied to a colorimage sequence acquired with a relatively inexpensive digital camera ofthe consumer product variety used in Web interactive applications (alsoknown as a webcam). In that case, the LR color image sequence 404 wasrecorded while a person held the camera in his hand for about one second(resulting in a sequence of frames being captured). The natural shakingof the user's hand provided the necessary motion for obtaining differentsampling grids in the LR images. As can be seen in FIG. 5, the image isa linear interpolation (by a factor of ×3) of the three color channels(to match the higher resolution) from a single LR frame, whereas theimage in FIG. 6 is the HR reconstruction obtained by the SR method forcolor images described above, where in this case a generic Huberfunction was used for the Likelihoods and Priors. It is evident that theresulting HR image contains much more detail than the interpolatedimage.

Point Spread Function Calibration

Recall that the point spread function (PSF) models the non-ideality ofthe camera (also referred to as an image acquisition system). Although aprecise knowledge of the PSF of an image acquisition system may not becritical for SR methods to work, the quality of the result HR image maybe further improved if such knowledge is incorporated into the SRmethod. A PSF may be theoretically computed based on the specificationsof the image acquisition system. For example, in a video charge coupleddevice (CCD) camera, the lens and the CCD sensor specification may beused to compute the PSF. However, that information is not alwaysavailable, in which case the PSF is estimated by calibration.

An existing method to estimate the PSF is to obtain an image thatcorresponds to a punctual source (e.g., a white point on a blackbackground). Alternatively, the image may correspond to an equivalentpunctual source, such as an expanded laser beam. The image thusprojected in the image plane (focal plane) of the camera sensorcorresponds to the PSF. This optical image is sampled by the sensor, toobtain a digital version. If the sampling frequency is higher than twicethe highest frequency of the PSF, then the digital version may beconsidered a complete representation of the underlying, continuous PSF.However, in the case of super resolution reconstruction, the samplingfrequency (for the LR images) is clearly lower than the one needed toavoid aliasing. Therefore, a single, LR image of a punctual source is anoisy and potentially aliased version of the underlying PSF.

According to an embodiment of the invention, a higher resolution,aliasing free version of the PSF is recovered using an LR image sequenceof a moving punctual source, instead of a single image. This method maybe essentially the same as the ones described above for obtaining an HRimage from an LR image sequence, except that in this case the processhas the knowledge that the result HR image is that of a punctual source,and also that the PSF is not known. Since there is a linear relationbetween a punctual source and a PSF, it is possible to interchange theroles of the scene being imaged and the PSF. Thus, to recover the PSF,it may be sufficient to apply the same SR method described above to animage sequence obtained using the punctual source, with the PSF as apoint (or, more generally, the known images used as a test forcalibrating the PSF). The recovered HR image should be a higherresolution version of the underlying PSF. This resulting, calibrated PSFmay then be used in the observation model, for determining theLikelihood function in the SR methods described earlier.

System Applications

The SR methods described above may be used in a variety of differentsystem applications, provided there is enough computational power toproduce a solution to the estimation process in a reasonable time. Assmall and inexpensive digital image acquisition devices are becomingcommon place, such as consumer grade digital cameras and webcams, the SRmethods may be implemented using LR images captured by such devices, toprovide enhanced digital images from limited image acquisition hardwarecapability. Specific examples include resolution improvement in imagesacquired with solid state digital cameras attached to cellular/mobiletelephones, personal digital assistants, and other small electronicdevices whose main purpose is not to acquire images. In suchapplications, a sequence of LR images are captured while the camera isbeing held by the user, where the natural motion of the user's hand willproduce the motion needed to generate the needed LR images. Suchportable devices may, however, lack the computational power to executethe operations required by SR methods in a reasonable time. The LR imagesequence could instead be transmitted to either a dedicated server thatprovides computing services (such as a Web based service business model)for this particular application, or to a personal computer in which theHR image or image sequence may be reconstructed.

With respect to webcams, again their primary purpose may not be to takehigh resolution images. Accordingly, the SR methods will convert thisrelatively inexpensive, low resolution device into a high resolutioncamera. For example, the increase in resolution may allow a webcam witha standard video graphics resolution of 640×480 to scan a letter sizeddocument at a resolution of 200 dots per inch, suitable for printing andfax transmission at reasonable quality. This inexpensive and relativelycommon device may then be used as an occasional document scanner, bysimply placing the document to be scanned on the user's desk and aimingthe webcam at the document, taking a sequence of images while the useris holding the webcam above the document in her hand. No additionalequipment is needed to hold the camera, because the natural shaking ofthe user's hand provides the motion needed for differences between theLR images so that the super resolution method will work to yield a highresolution image.

In yet another application, resolution improvement may be achieved forconversion of standard video to high definition video. In that case, Nframes may be collected from time t to time t+N (in frames), where theseframes become the LR images used to generate the high resolution framecorresponding to time t+N. In this case, the resolution improvement maybe limited to the part of a scene that is visible during the interval inwhich the low resolution frames are collected. This resulting HR framewill be a clear perceptual improvement with respect to a simpleinterpolation of the standard video to high definition video. Thisembodiment may be used to generate, for example, high definitiontelevision, HDTV, video from standard video sequences, or to generate HRimages that are suitable for high definition printing from standard(lower resolution) video sequences.

The SR methods may also be applied to obtain image enhancement,including de-noising, de-blurring, and resolution improvement, in imagesthat have been acquired with scanning imaging devices (e.g., scanningelectron microscope, focused ion beam, and laser voltage probe). Toobtain the different LR images needed for the SR method, these scanningimaging devices allow the scanning pattern to be varied, thus producingdifferent sampling grids with sub-pixel shifts needed for the SR method.Such devices may be part of tools used in microelectronic test andmanufacturing, to image and/or repair semiconductor structures andlithography masks. In some cases, such tools need to be operated at alower resolution than the maximum possible, to increase throughput orbecause the parameters of the tool are optimized for nano-machiningrather than optimal imaging. With such images, specific Prior models maybe available that can be adapted to render the SR methods moreeffective.

Also, as microelectronic manufacturing advances, the features of thestructures being inspected are becoming smaller and smaller, such thatlower quality images may be produced in the future when using currentscanning imaging devices. By enhancing images from older generationscanning imaging devices, the life span of such tools will be extendedin the future, without having to upgrade or replace the tools, therebytranslating into significant savings in tooling costs. FIGS. 7-9 and10-11 show two examples, respectively of applying the SR method toreconstruct a high resolution scanning imaging device image. In thefirst example (FIGS. 7-9), a high resolution focused ion beam image isto be reconstructed, from a simulated noisy low resolution millingsequence. In FIG. 7, an original HR image acquired with a focused ionbeam tool is shown. In FIG. 8, one LR image out of a sequence of 4×subsampled images after low pass filtering, with additive noise isshown. FIG. 9 shows the SR reconstruction. Note the clear improvement indetail between the SR reconstruction (FIG. 9) and the LR image (FIG. 8).The improvement in detail is also apparent in the second example,corresponding to a real milling sequence with displaced millboxes.Compare one of the initial LR images (FIG. 10), magnified ×8 usingnearest neighbor interpolation, and the result HR image after applyingSR reconstruction, magnified ×8 (FIG. 11).

The SR methods described above may be implemented using a programmedcomputer. A computer program product or software may include a machineor computer-readable medium having stored thereon instructions which maybe used to program a computer (or other electronic devices) to perform aprocess according to an embodiment of the invention. In otherembodiments, operations might be performed by specific hardwarecomponents that contain microcode, hardwired logic, or by anycombination of programmed computer components and custom hardwarecomponents.

A machine-readable medium may include any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer), but is not limited to, floppy diskettes, optical disks,Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks,Read-Only Memory (ROMs), Random Access Memory (RAM), ErasableProgrammable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), magnetic or optical cards, flashmemory, a transmission over the Internet, electrical, optical,acoustical or other forms of propagated signals (e.g., carrier waves,infrared signals, digital signals, etc.) or the like.

The invention is not limited to the specific embodiments describedabove. For example, the noise n in the observation model of Equation(1), which is modeled as a non-Gaussian robust function, mayalternatively be any noise distribution previously learned from pairs ofHR images and LR image sequences. Accordingly, other embodiments arewithin the scope of the claims.

1. A method for image processing, comprising: computing a result higherresolution (HR) image of a scene given a plurality of observed lowerresolution (LR) images of the scene using a Bayesian estimation imagereconstruction methodology, wherein the methodology yields the result HRimage based on a Likelihood probability function that implements a modelfor the formation of LR images in the presence of noise, and wherein themethodology models the noise by a probabilistic, non-Gaussian, robustfunction.
 2. The method of claim 1 wherein the methodology yields theresult HR image based on maximizing posteriori probability, that is theconditional probability of an unknown HR image given the observed LRimages, wherein the methodology yields the result HR image based oncombining the Likelihood function with a Prior probability function, thePrior function indicating which HR images are probable.
 3. The method ofclaim 2 wherein the modeling of the noise by the robust function causesthe role of a statistical outlier pixel in an observed LR image to bedownplayed when computing a trial HR image based on the Likelihoodfunction, so that a computed Likelihood probability for said observed LRimage is higher than if the noise were modeled by a Gaussian function.4. The method of claim 3 wherein: the robust function is a Huberfunction.
 5. The method of claim 4 wherein: the robust function is aTukey function.
 6. The method of claim 2 wherein the Prior function usedin the methodology implements one of a Gaussian Markov Random Field(MRF), a Huber MRF, and Tukey MRF to indicate the probability of whichpixels in an image take on which values.
 7. The method of claim 3further comprising: setting the robust function to a different functiondepending on whether the observed LR images have relatively few andrelatively many outliers, wherein the methodology estimates a shape ofthe robust function according to the availability of training data. 8.The method of claim 3 wherein the methodology estimates a shape of therobust function by selecting a scale factor, wherein if there issufficient training data in the form of one or more ground truth HRimages and their corresponding LR images, the scale factor is estimatedfrom samples obtained in computing an error between the observed LRimages of the scene and their projections from the ground truth HRimages.
 9. The method of claim 8 wherein if there is insufficienttraining data, the scale factor is estimated by (1) taking a current,trial HR image of an iterative, maximum a posteriori estimation processas a ground truth HR image, and (2) a robust estimator for the scalefactor.
 10. The method of claim 3 further comprising defining the Priorfunction based upon computer aided design models for structures capturedin the observed LR images.
 11. The method of claim 3 wherein the Priorfunction is based on a non-Gaussian, robust Markov Random Field.
 12. Asystem comprising: a processor; and memory having instructions that,when executed by the processor, generate a result higher resolution (HR)image of a scene based on a plurality of lower resolution (LR) images ofthe scene, using a Bayesian image reconstruction methodology based on aLikelihood probability function that implements a model for LR imageformation that includes additive noise, and wherein the methodologymodels the additive noise by a probabilistic, non-Gaussian, robustfunction.
 13. The system of claim 12 wherein the processor and memoryare part of one of a desktop and notebook personal computer, and whereinthe memory stores further instructions that when executed by theprocessor obtain the plurality of LR images based on images downloadedinto the personal computer from a digital camera.
 14. The system ofclaim 13 wherein the instructions are to obtain the plurality of LRimages as video, based on video downloaded into the personal computerfrom the digital camera, and wherein a plurality of result HR images areto be generated as HR video of the scene.
 15. The system of claim 14wherein the instructions are to generate the HR video in a highdefinition television, HDTV, format.
 16. The system of claim 12 whereinthe instructions are to yield the result HR image based on maximizingposteriori probability which is a combination of the Likelihood functionand a Prior probability function, the Prior function indicating which HRimages are probable.
 17. An article of manufacture comprising: a machineaccessible medium containing instructions that, when executed, cause amachine to compute a result higher resolution (HR) image of a scenegiven a plurality of observed lower resolution (LR) images of the sceneusing a Bayesian image reconstruction methodology, wherein themethodology yields the result HR image based on a Likelihood probabilityfunction that implements a model for LR image formation in the presenceof noise, and wherein the methodology models the noise by a weightingfunction that causes the role of a statistical outlier pixel in anobserved LR image to be downplayed when computing a trial HR image basedon the Likelihood function, so that a computed Likelihood probabilityfor said observed LR image given the trial HR image is higher than ifthe noise were modeled by a Gaussian function.
 18. The article ofmanufacture of claim 17 wherein the instructions are such that themethodology yields the result HR image based on maximizing posterioriprobability, that is the conditional probability of an unknown HR imagegiven observations about LR images, wherein the methodology yields theresult HR image based on combining the Likelihood function with a Priorprobability function, the Prior function indicating which HR images areprobable.
 19. The article of manufacture of claim 18 wherein the mediumincludes further instructions that set the weighting function to adifferent function depending on whether the plurality of observed LRimages have relatively few and relatively many outliers, and wherein themethodology estimates a shape of the weighting function according to theavailability of training data.
 20. The article of manufacture of claim18 wherein the instructions are such that the methodology estimates ashape of the weighting function by selecting a scale factor, wherein ifthere is sufficient training data in the form of one or more groundtruth HR images and their corresponding LR images, the scale factor isestimated from samples obtained in computing an error between theplurality of observed LR images of the scene and their projections fromthe ground truth HR images.
 21. The article of manufacture of claim 20wherein the instructions are such that if there is insufficient trainingdata, the scale factor is estimated by (1) taking a current, trial HRimage of an iterative, maximum a posteriori estimation process as aground truth HR image, and (2) a robust estimator for the scale factor.